Enforcing Grammar in Code Synthesis with Transformers

Even more so than natural language, code is extremely sensitive to syntax; a small error could make an entire snippet invalid. It therefore important explore methods for ensuring syntactic correctness in generated code. Existing resolve this issue often rely on the complex architecture of syntax-guided decoders. In work, we present grammar enforcement method, which introduces separate layer that constrains decisions transformer during fine-tuning according constructs both target language and given training set. We experiment with Hearthstone dataset study its effects result programs compare it existing state-of-art demonstrate statistically significant positive effect quality terms exact match accuracy grammatically correct percent samples. At same time, observe lower values text-based metrics, chrF, BLEU, potentially indicating their inability represent abstract syntax sequences.